Distributed Machine Learning: Foundations, Trends, and Practices

In recent years, artificial intelligence has achieved great success in many important applications. Both novel machine learning algorithms (e.g., deep neural networks), and their distributed implementations play very critical roles in the success. In this tutorial, we will first review popular machine learning algorithms and the optimization techniques they use. Second, we will introduce widely used ways of parallelizing machine learning algorithms (including both data parallelism and model parallelism, both synchronous and asynchronous parallelization), and discuss their theoretical properties, strengths, and weakness. Third, we will present some recent works that try to improve standard parallelization mechanisms. Last, we will provide some practical examples of parallelizing given machine learning algorithms in online application (e.g. Recommendation and Ranking) by using popular distributed platforms, such as Spark MlLib, DMTK, and Tensorflow. By listening to this tutorial, the audience can form a clear knowledge framework about distributed machine learning, and gain some hands-on experiences on parallelizing a given machine learning algorithm using popular distributed systems.